We sincerely thank the anonymous reviewers for their support and constructive comments. We will refine the presentation and add a dedicated section for revisiting related-works. More results will be included in the final version. We will refine the presentation around the definitions. These methods re-train the original network to learn the importance of filters.

We sincerely thank the anonymous reviewers for their supports and constructive comments. Q1: Compare with vanilla ANN with dropout. The knowledge from CNN will help ANN to find the correct optimization direction. The experimental results on CIFAR-10 are shown in Table 1 column 2 and 4. Q2: Does PKKD change the inference process? We will further emphasize this in the final version.